Aggregating Congestion Information Over Sequences of TCP Connections

In this paper we present an extension of the TCP stack that allows a sequence of TCP connections between the same machines to share the congestion window. Our Linux implementation of this scenario shows significant improvement in performance, particularly when the individual connections are short-lived. Such a behavior is common on the web, due to the nature of the HTTP protocol and the distribution of file sizes. Summary: 1. Motivation 2. Measuring procedure 3. Results 4. Future developments 5. Implementation 6. Code 1. Motivation One of the essential features of TCP is the sliding window algorithm. The congestion window size defines the amount of data, in segments, that a TCP sender can transmit before it has to wait for an acknowledgment to proceed. It provides the Transport level with a flow control mechanism (end-to-end) and it makes sure that data is correct and delivered in the right order. During a TCP connection, the window size is adjusted according to the receiver's ability to handle incoming TCP segments, which essentially depends on its memory resources (and CPU speed). It starts with a congestion window size of 1 segment that grows with time if everything works well. This mechanism works over one TCP connection, but the size of the window starts again with its initial 1 segment value for every new connection. We know that an HTTP session is based on opening and closing numerous very short TCP connections between a client and a 5/10/12 Remembering the Congestion Window Size over Several TCP Connections 2/21 www.cs.bu.edu/techreports/pdf/1998‐001‐aggregate‐tcp‐sequ.pdf/ server. Given that, an HTTP session cannot take advantage of the sliding window mechanism, since it will rarely go further than a few round-trips, which does not allow the window size to grow significantly, even though resources might be available to increase efficiency. We want to keep track of the TCP congestion window size over several connections between two computers, and over a given period of time. This way, the regular mechanism could go further than only a few steps and larger window sizes could be used, decreasing overhead and therefore improving efficiency. 2. Measuring procedure To check the validity of the above idea, we needed to set up a way of measuring and visualizing the gain we would achieve over a traditional implementation of TCP. The mechanism remembering the congestion window size over several TCP connections was implemented on a machine that would simulate an HTTP server. Another machine had to simulate an HTTP client. Since local measurements, e.g. using computers hooked up to the same LAN, proved not to be appropriate because the round-trip time was so short that the gain was very small, we installed the client part in Paris, France, with the server machine located in Boston, MA. But this did not allow to see what would happen for intermediate distances. In order to simulate a variety of distances, a local client machine was modified so that it introduced a fixed delay every time a TCP connection acknowledged a packet. Thus we measured the behavior of the new mechanism both on one long distance connection and on a local connection simulating wider ones with delays. Application tools were developed in order to simulate an HTTP server on the server machine and an HTTP client on the "delayed" machine: respectively httpsims and httpsimc. httpsims, which syntax is httpsims , waits for incoming connections on a TCP port, sends bytes and closes the connection. httpsimc, which syntax is httpsimc , opens a connection on of , receives data from the server application, closes the connection and starts all over again. It keeps on opening a TCP connection, receiving data and closing the connection for seconds. To measure the actual behavior of TCP we used the tcpdump UNIX command, which prints out the headers of packets on a network interface. Used together with conv, an extra program written to process its output, it provides one with data that can be plotted and that gives a better view of what has been achieved (see implementation details and code in sections 5.4 and 6.4). 5/10/12 Remembering the Congestion Window Size over Several TCP Connections 3/21 www.cs.bu.edu/techreports/pdf/1998‐001‐aggregate‐tcp‐sequ.pdf/ 3. Results This procedure led to the following four graphs. Figure 1 compares the behavior of TCP with our mechanism with the behavior of the conventional TCP and clearly shows that we benefit from remembering the congestion windows size over several TCP connections. Figure 1 However, this was done using only local measurements. Figure 2 shows the same thing on a "real" long distance connection (Boston/Paris). Figure 2 To give a more accurate idea of the speedup in relation to the distance, Figure 3 shows the speedup related to the delays we introduced in order to simulate various distances. It shows it for four different amounts of data transmitted, ranging from 10 MB to 40 MB. We see that we get the best speedup for the biggest delay � the widest simulated network � and that the gain is lower for bigger amounts of transmitted data. Our mechanism's performance is at its best when the round-trip time gets longer. Figure 3 Figure 4 demonstrates better what actually happens to the congestion window size. It shows the evolution of the congestion window size over time with and without the remembering mechanism: the regular TCP behavior is cyclic over several connections whereas the new implementation keeps increasing the congestion window size. 5/10/12 Remembering the Congestion Window Size over Several TCP Connections 4/21 www.cs.bu.edu/techreports/pdf/1998‐001‐aggregate‐tcp‐sequ.pdf/ Figure 4 4. Future developments A few ideas: The way stored congestion window sizes are eventually discarded (see section 5.1 below) is quite raw and could be refined. We could use a decay function based on some "forgetfulness factor" computed with data collected during previous connections in order to decide when to stop using a window size. Congestion window sizes are collected only every time a TCP connection ends. Collecting them throughout a connection would be more robust and might be required to compute the above decay function. 5. Implementation 5.1 Implementation of the Congestion Window Size Remembering mechanism This was done on the server machine. In Linux, the congestion window size is stored in the variable cong_window in the sock structure (in linux/include/net/sock.h). A linked list was implemented in kernel memory with a set of functions to store and search congestion window sizes. Three pieces of information are saved: the IP address of the peer computer; the size of the congestion window; a Linux time stamp (seconds since 1/1/1970). Two constants were defined to adjust the behavior of the system: LIFETIME_WIND: for how many seconds should we keep window sizes in memory? MAX_WINDLIST: how many of them do we want to store? If the system reaches MAX_WINDLIST, a garbage collector function deletes all congestion window sizes that are older than LIFETIME_WIND in order to make more room. Also, whenever we find out during a search that the window size we want is outdated, we do not 5/10/12 Remembering the Congestion Window Size over Several TCP Connections 5/21 www.cs.bu.edu/techreports/pdf/1998‐001‐aggregate‐tcp‐sequ.pdf/ use it, we delete it. When the system boots, it behaves just like any Linux system. The capture of window sizes can be started and stopped manually using this trick: various operations implemented in the kernel will be performed whenever TCP sees some specific IP address. Therefore we can telnet to these addresses in order to get these operations completed: telnet cs1 starts the capture; telnet cs2 stops the capture (and frees the linked list). Congestion window sizes must be collected every time a TCP connection is closed. This is done in tcp_close() and tcp_shutdown() (in linux/net/ipv4/tcp.c). Only the biggest window size among those used for several consecutive connections between two computers is stored: this value cannot decrease (it only can disappear eventually after LIFETIME_WIND). Congestion window sizes must be used instead of the basic initial size (1 packet) for every incoming TCP connection. This is done in tcp_conn_request() (in linux/net/ipv4/tcp_input.c). (See code in section 6.1) 5.2 Implementation of an extra feature that displays captured window sizes Getting data out of kernel space is not an easy job. Conveniently, the Proc file system provides a way of accessing kernel data from user space. Kernel variables such as max_files are already available by accessing various pseudo-files in /proc (for instance /proc/sys/kernel/file-max for max_files). Creating a new entry � a new pseudo-file in /proc � is simple: we just need to update a few structures and functions in linux/include/linux/proc_fs.h, linux/fs/proc/root.c and linux/fs/proc/array.c and write a function that will fill in a page with what we want to display whenever invoked. The entry added in /proc for this project is called project and the function it calls is get_project_status() (implemented in linux/net/ipv4/tcp.c). To display the linked list (IP address, window size, age), we type: